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Abstract 

This  paper  provides  information  on  the  reliability  and  validity  of  an  employee 
questionnaire  developed  in  a  study  of  patient  safety  in  outpatient  surgery.  The 
Systems  Engineering  Intervention  in  Outpatient  Surgery  (SEIPS),  a  collaborative 
community  perspective  project  currently  underway  at  the  University  of 
Wisconsin-Madison,  examines  the  impact  of  a  systems  engineering  intervention 
on  both  employees  and  patients.  In  this  paper,  we  describe  the  SEIPS  employee 
questionnaire,  which  surveys  various  elements  of  the  work  system  (e.g., 
communication,  workplace,  supplies,  and  patient  safety  climate),  the  care  process, 
and  employee  outcomes  (e.g.,  job  satisfaction,  stress,  perceived  quality  and  safety 
of  care  provided).  Data  from  a  sample  of  289  staff  members  in  5  outpatient 
surgery  centers  (53  percent  response  rate)  are  used  to  examine  reliability, 
construct  validity,  convergent  validity,  and  predictive  validity.  The  results 
provided  evidence  for  the  reliability  and  validity  of  the  SEIPS  study’s  employee 
questionnaire. 


Introduction 

The  Systems  Engineering  Intervention  in  Outpatient  Surgery  (SEIPS),  a 
collaborative  community  perspective  project  currently  underway  at  the  University 
of  Wisconsin-Madison,  is  applying  the  SEIPS  work  system  model  to  patient 
safety  in  outpatient  surgery  centers.1  The  objective  of  the  project  is  to  examine  the 
impact  of  a  system  intervention  on  the  work  system,  employee  and  organizational 
outcomes,  and  quality  and  safety  of  patient  care.  The  five  major  outpatient 
surgery  centers  in  Madison  are  cooperatively  participating  in  the  study.  All  five 
sites  are  members  of  the  Madison  Patient  Safety  Collaborative,  a  group  of  local 
providers  committed  to  improving  patient  safety  in  the  community  who  have 
agreed  to  not  use  patient  safety  to  their  competitive  advantage.  In  the  course  of 
this  research  study,  all  employees  working  in  the  five  outpatient  surgery  centers 
were  asked  to  voluntarily  respond  to  an  employee  survey,  designed  to  assess 
patient  safety  in  outpatient  surgery  from  the  viewpoint  of  the  employees.  In  this 
paper,  we  describe  the  steps  used  to  develop  the  content,  reliability,  and  validity 
of  the  survey. 

There  is  a  conceptual  framework  underlying  the  structure  of  the  employee 
questionnaire.  In  the  SEIPS  model  of  work  system  and  patient  safety,  we 
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integrated  Donabedian’s  structure-process-outcome  framework  and  the  work 
system  model.3, 4  The  structure  of  an  organization — or,  more  generally,  the  work 
system — affects  the  care  process,  and  the  means  of  caring  for  and  managing  the 
patient  (the  care  process)  affects  patient  safety  (patient  outcome)  and  employee 
and  organizational  outcomes.1  The  employee  questionnaire  asks  about  various 
elements  of  the  work  system  (e.g.,  communication,  workplace,  supplies,  and 
patient  safety  climate),  the  care  process,  and  employee  outcomes  (e.g.,  job 
satisfaction,  stress,  perceived  quality  and  safety  of  care  provided). 


Methods 

Study  setting 

In  2002,  representatives  of  the  five  outpatient  surgery  centers  discussed  a  joint 
research  effort  aimed  at  addressing  quality  and  safety  issues  at  their  respective 
and  combined  centers.  The  “pilot  team”  of  SEIPS  researchers  and  representatives 
from  each  of  the  centers  agreed  that  a  baseline  had  to  be  established  to  determine 
the  focus  of  the  intervention  they  would  ultimately  implement  at  each  center. 
Through  an  open-ended  survey,  medical,  nursing,  technical,  and  clerical  staffs 
were  asked  to  identify  quality  and  safety  issues  at  their  centers,  as  well  as  working 
conditions  issues  that  either  interfered  with  or  facilitated  their  ability  to  perform 
their  jobs.5,6 

From  the  information  collected  with  the  initial  employee  questionnaire,  the 
pilot  team  agreed  that  each  center  could  choose  its  own  specific  intervention,  but 
that  any  intervention  should  address  “coordination  and  communication  of  care” 
between  providers  prior  to  outpatient  surgery.7  The  interventions  were  evaluated 
using  two  data  collection  instruments:  an  employee  questionnaire  and  a  patient 
telephone  survey.  Data  were  collected  prior  to  the  implementation  of  the 
intervention  and  again  12  months  later.  In  this  paper,  we  describe  the  employee 
questionnaire  tool  that  we  developed  to  evaluate  the  systems  engineering 
intervention  and  report  baseline  data  that  were  collected  before  implementation. 

Sample 

Employees  (physicians,  nurses,  and  other  technical  and  administrative 
personnel)  from  the  five  outpatient  surgery  centers  in  Madison,  Wisconsin,  were 
asked  to  complete  the  questionnaire.  A  total  of  53 1  questionnaires  were 
distributed  to  the  surgery  centers’  staffs;  289  surgery  center  employees 
participated,  yielding  an  overall  response  rate  of  53  percent  (ranging  from  48-91 
percent).  Thirty-six  percent  of  the  respondents  were  physicians  (60  percent  of 
whom  were  surgeons,  40  percent  anesthesiologists),  and  38  percent  were  nurses. 
The  rest  of  the  sample  included  technicians,  office  personnel,  and  schedulers. 
Women  represented  67  percent  of  the  sample.  The  age  distribution  was  as 
follows:  younger  than  25:  2  percent;  25-34  years:  16  percent;  45-54  years:  31 
percent;  55-64  years:  43  percent;  and  older  than  65:  1  percent.  The  majority 
worked  either  3 1-40  hours  per  week  (41  percent)  or  more  than  41  hours  per  week 
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(40  percent).  Tenure  with  current  employer  was  9.6  years  on  average  (standard 
deviation  [SD]  =  7.5  years),  and  the  average  number  of  years  in  current  job  was  8 
(SD  =  7.3  years).  Ninety-six  percent  of  respondents  were  white. 

Employee  questionnaire 

The  employee  questionnaire*  included  a  total  of  7 1  questions  covering  the 
following  domains: 

•  Work  system — 34  questions  on  communication  openness, 
communication  accuracy,  communication  timelines,  time  pressures 
affecting  patient  safety,  workload,  coordination  mechanisms, 
workplace  design,  equipment  design,  and  access  to  supplies. 

•  Patient  safety  climate — 17  questions,  2  of  them  were  answered  only  by 
physicians. 

•  Perceived  performance — 6  questions  on  unit  effectiveness  and 
satisfaction  with  care  provided. 

•  Quality  of  working  life — 5  questions  on  job  satisfaction,  fatigue,  and 
tension. 

•  Demographic  and  background  information — 9  questions. 

Table  1  provides  information  on  the  source  of  the  questions,  the  number  of 
questions  for  each  concept,  and  examples  of  the  questions.  Because  the  system 
intervention  implemented  in  the  outpatient  surgery  centers  focused  on  improving 
communication  and  coordination  within  the  centers,  as  well  as  between  the 
centers  and  other  units,  the  questions  on  the  work  system  included  in  the  survey 
specifically  focused  on  several  dimensions  of  communication  and  coordination. 

Study  procedures 

As  the  study  subjects  were  drawn  from  five  separate  outpatient  surgery 
centers  with  varying  case  mixes,  types  of  employees,  and  a  range  of  physical 
settings  (i.e.,  in-hospital  setting,  free-standing  clinics,  and  clinics  in  close 
proximity  to  the  hospital),  several  data  collection  procedures  were  used  for  each 
site.  The  survey  questionnaire  was  originally  formatted  in  Microsoft  Word®. 
However,  a  Web-based  format  was  offered  in  addition  to  the  paper-based 
questionnaire,  in  hopes  of  catering  to  the  needs  and  schedules  of  busy  surgical 
center  staff.  The  Microsoft  Word  paper  questionnaire  survey  was  converted  to  a 
Web-based  survey  using  Macromedia  Dreamweaver®  MX  Education  Version. 

The  distribution  of  the  personalized  e-mail  cover  letters,  the  electronic 
questionnaire  survey,  and  anonymous  survey  tracking  was  accomplished  using 
WSMS1.1®  Web  Survey  Mailer  System.8  The  choice  of  the  survey  format 
(electronic  or  paper)  for  distribution  was  ultimately  the  individual  decision  of 
each  surgical  center’s  top  management.  All  the  outpatient  surgical  centers,  with 
one  exception,  chose  to  distribute  the  paper  questionnaire  survey.  One  center 


A  copy  of  the  complete  employee  questionnaire  is  available  from  the  corresponding  author. 
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Table  1.  Description  of  the  variables 


Concept 

#  of  questions 

Source 

Example  of  question 

Communication/ 

openness 

4 

Shorten  et  al.a 

Communication  with  the  nursing 
staff  in  this  unit  is  very  open. 

Communication/ 

accuracy 

4 

Shorten  et  al.a 

1  can  think  of  a  number  of  times 
when  1  received  incorrect 
information  regarding  patient  care 
from  nurses  in  this  center. 

Communication 

/timeliness 

3 

Shorten  et  al.a 

1  get  information  on  the  status  of 
patients  when  1  need  it. 

Time  pressure 
affecting  patient 
safety 

2 

Singer  et  al.b 

1  have  enough  time  to  complete 
patient  care  tasks  safely. 

Workload 

1 

Caplan  et  al.c 

How  often  is  there  a  great  deal  to  be 
done? 

Staffing 

1 

Gray-Toft  and 
Andersond 

How  often  is  there  not  enough  staff 
to  adequately  cover  the  center? 

Coordination 

mechanisms 

5 

Shorten  et  al.a 

How  effective  are  written  rules, 
policies  and  procedures  for  the 
coordination  of  staff  activities? 

Workplace 

design 

6 

Shorten  et  al.a 

Your  workplace  is... 

...  unnecessarily  noisy  /  reasonably 
quiet 

Equipment 

design 

6 

Shorten  et  al.a 

The  equipment  you  work  with  is. . . 

...  modern  /  outdated 

Access  to 
supplies 

2 

Shorten  et  al. a 

Supplies  provided  for  your  use 
are... 

. . .  usually  out  of  stock  /  available 
when  needed 

Patient  safety 
climate 

17  (2  questions 
answered  only 
by  physicians) 

Singer  et  al.;b 
Gaba  et  al.;e 

Nieva  and  Sorraf 

1  feel  that  it  is  just  pure  luck  that 
more  serious  mistakes  don’t  happen 
around  here. 

Unit 

effectiveness 

5 

Shorten  et  al.a 

Given  our  patient  population  and  the 
procedures  that  we  perform,  our 
center’s  patients  experience  very 
good  outcomes. 

Satisfaction  with 
care  provided 

1 

Bertram  et  al.9 

In  general,  1  am  satisfied  with  the 
quality  of  care  that  1  provide. 

Job  satisfaction 

1 

Quinn  et  al.h 

All  in  all,  how  satisfied  would  you 
say  you  are  with  your  job? 

Fatigue 

2 

McNair  et  al.1 

How  have  you  been  feeling  during 
the  past  week . fatigued 

Tension 

2 

McNair  et  al.1 

How  have  you  been  feeling  during 
the  past  week . nervous 

a  Shorten  SM,  Rousseau  DM,  Gillies  RR,  et  al.  Organizational  assessment  in  intensive  care  units 
(ICUs):  construct  development,  reliability,  and  validity  of  the  ICU  nurse-physician  questionnaire. 
Med  Care  1991  Aug;29(8):709-27. 

b  Singer  SJ,  Gaba  DM,  Geppert  JJ,  et  al.  The  culture  of  safety:  results  of  an  organization-wide 
survey  in  15  California  hospitals.  Qual  Saf  Health  Care  2003  Apr;  1 2(2):  1 12-8. 
cCaplan  RD,  Cobb  S,  French  JR,  et  al.  Job  demands  and  worker  health;  main  effects  and 
occupational  differences.  Cincinnati:  National  Institute  for  Occupational  Safety  and  Health;  1975. 
d  Gray-Toft  P,  Anderson  JG.  The  nursing  stress  scale:  development  of  an  instrument.  J  Behav 
Assess  1981  ;3(1):1 1-23. 

eGaba  DM,  Howard  SK,  Jump  B.  Production  pressure  in  the  work  environment.  California 
anesthesiologists'  attitudes  and  experiences.  Anesthesiology,  1994  Aug;81(2):488-500. 
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Table  1.  Description  of  the  variables,  cont. 

f  Nieva  VF,  Sorra  J.  Safety  culture  assessment:  a  tool  for  improving  patient  safety  in  healthcare 
organizations.  Qual  Safe  Health  Care  2003  Dec;12(Suppl  2):ii1 7—23. 

3  Bertram  DA,  Hershey  CO,  Opila  DA,  et  al.,  A  measure  of  physician  mental  work  load  in  internal 
medicine  ambulatory  care  clinics.  Med  Care  1990  May;28(5):458-67. 

h  Quinn  R,  Seashore  S,  Kahn  R,  et  al.  Survey  of  working  conditions:  final  report  on  univariate  and 
bivariate  tables.  Washington,  DC:  U.S.  Government  Printing  Office;  1971.  Document  No. 2916- 
0001. 

'McNair  DM,  Lorr  M,  Droppleman  LF.  EITS  manual  for  profile  of  mood  states.  San  Diego,  CA: 
Educational  and  Industrial  Testing  Service;  1971. 


distributed  the  Web-based  survey  to  its  physicians,  reasoning  that  ease  of 
computer-access  surveys  might  increase  the  number  of  surveys  completed  by  this 
group. 

A  member  of  the  research  team  introduced  and  explained  the  outpatient 
surgery  research  project  at  each  outpatient  surgery  centers’  regularly  scheduled 
staff  meetings.  Upon  completion  of  the  project  presentation  and  discussion,  the 
investigators  explained  and  distributed  the  employee  questionnaire  to  all  surgery 
center  staff.  Attendance  was  taken  at  each  site’s  staff  meeting,  and  questionnaires 
were  left  for  those  absent  employees  to  complete.  The  surgery  centers’ 
supervisory  staff  explained  and  distributed  questionnaires  to  the  absent  employees 
at  a  later  time.  An  institutional  review  board-required  cover  letter  and  information 
sheet,  explaining  the  research  project,  accompanied  each  questionnaire.  The 
surveys  were  returned  via  a  locked,  secure  drop  box  at  any  of  the  five  sites,  the 
U.S.  Postal  Service,  university  interdepartmental  mail,  or  via  the  Internet,  if 
subjects  chose  to  complete  the  Web  survey.  All  nonphysician  staff  were  given 
work  time  to  complete  and  return  the  questionnaires.  Completed  questionnaires 
were  retrieved  from  each  site’s  locked  drop  box  twice  weekly  by  a  research 
assistant  or  placed  in  self-addressed,  sealed  envelopes  and  returned  to  the 
principal  investigator. 

Data  analysis 

In  this  paper,  we  report  data  on  reliability  and  validity  for  the  measures  of 
work  system,  perceived  performance,  and  quality  of  working  life.  The  analysis  of 
reliability  and  validity  of  the  patient  safety  climate  is  reported  elsewhere.9 
Cronbach’s  alpha  scores  were  used  to  assess  the  reliability  of  scales.  We 
conducted  confirmatory  factor  analysis,  using  structural  equation  modeling  with  a 
maximum  likelihood  procedure,  in  order  to  assess  construct  validity.  This  analysis 
was  performed  with  the  AMOS®  software.  For  convergent  validity,  we  compared 
answers  to  various  scales  and  questions  among  three  job  categories:  nurses, 
physicians,  and  other  staff.  In  order  to  evaluate  predictive  validity,  we  examined 
the  relationship  between  measures  of  the  work  system  and  the  measures  of 
perceived  performance  and  quality  of  working  life.  This  analysis  was  based  on  a 
correlational  analysis  and  a  series  of  stepwise  regression  analysis. 
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Results 

Table  2  displays  basic  statistics  of  the  variables.  The  variables  had  adequate 
range,  and  the  Cronbach’s  alpha  scores  of  most  scales  were  satisfactory  (above 
0.70).  The  scales  of  communication  timeliness  and  access  to  supplies  both  had  a 
Cronbach’s  alpha  score  of  0.67. 


Table  2.  Basic  statistics  of  the  variables 


Scale 

No.  of 
items 

Mean 

(SD) 

Range 

Cronbach’s 

alpha 

score 

Coding 
(meaning  of 
high  score) 

WORK  SYSTEM 

Communication  openness 

4 

4.06 

(.62) 

1 .75-5 

.80 

High  comm, 
openness 

Communication  accuracy 

4 

2.07 

(.75) 

1-4.25 

.88 

Low  comm, 
accuracy 

Communication  timeliness 

3 

3.32 

(.42) 

2.33-5 

.67 

High  comm, 
timeliness 

Time  pressure  affecting 
patient  safety 

2 

4.01 

(.73) 

1 .40-5 

.76 

Low  time 
pressure 

Workload 

1 

3.80 

(1.07) 

1-5 

N/A 

High  workload 

Staffing 

1 

2.62 

(1.32) 

1-5 

N/A 

High  staffing 
problem 

Coordination  effectiveness 

5 

3.76 

(.68) 

1 .60-5 

.76 

High 

effectiveness 

Workplace  design 

6 

3.06 

(.79) 

1-5 

.78 

Little  workplace 
problem 

Equipment  design 

6 

3.77 

(.74) 

1 .83-5 

.83 

Little  equipment 
problem 

Access  to  supplies 

2 

3.81 

(.90) 

1-5 

.67 

Little  supplies 
problem 

PERCEIVED 

PERFORMANCE 

Unit  effectiveness 

5 

4.23 

(.54) 

1 .40-5 

.81 

High 

effectiveness 

Satisfaction  with  care 
provided 

1 

4.42 

(.61) 

1-5 

N/A 

High 

satisfaction 

QUALITY  OF  WORKING 

LIFE 

Job  satisfaction 

1 

1.41 

(.54) 

1-4 

N/A 

Low  satisfaction 

Fatigue 

2 

2.21 

(.97) 

1-5 

.84 

High  fatigue 

Tension 

2 

1.65 

(.81) 

1-5 

.79 

High  tension 

SD  =  standard  deviation 
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The  initial  step  of  the  confirmatory  factor  analysis  involved  the  replacement 
of  all  missing  data  (some  questions  were  not  answered  by  some  people)  for  each 
variable  by  the  mean  on  that  variable.  The  highest  number  of  missing  data  values 
was,  in  eight  cases,  for  the  following  question:  “How  effective  are  computerized 
information  systems  to  the  coordination  of  staff  activities?”  The  confirmatory 
factor  analysis  involved  two  steps:  (1)  confirmatory  factor  analysis  of  each  of  the 
scales  in  order  to  find  out  whether  the  a  priori  model  (concept)  fits  the  data,  and 
(2)  a  second-order  factor  analysis  on  two  groups  of  scales  (scales  of 
communication  and  scales  on  environment/equipment/supplies)  to  find  out 
whether  each  group  of  scales  could  be  represented  by  only  one  underlying  factor. 
This  second  procedure  has  as  an  additional  benefit  as  compared  to  the  first  step 
because  information  can  be  retrieved  from  the  scales  (models)  for  which 
insufficient  information  is  available  (the  models  are  “underidentified”).  Table  3 
shows  the  results  of  the  confirmatory  factor  analysis.  In  the  analysis  we  allowed 
for  covariance  between  items  that  belong  to  the  same  scale,  but  not  for  covariance 
between  items  that  belong  to  different  scales. 

The  results  show  that,  in  general,  the  models  fit  the  data  well,  although  not 
always  optimally.  Results  of  the  analysis  of  the  three  communication  scales  (i.e., 
communication  openness,  communication  accuracy,  and  communication 
timeliness)  show  that  a  model  with  one  underlying  factor  fits  the  data  slightly 
better  than  a  model  with  three  factors.  The  results  of  the  analysis  of  the  scales  on 
environment/equipment/supplies  show  that  three  underlying  factors  fit  the  data 
much  better  than  a  model  with  only  one  underlying  factor.  The  results  show  that 
most  of  the  models  could  not  be  improved  by  allowing  for  covariance  between 
items  that  do  not  belong  to  the  same  scale.  This  is  an  indication  of  the  construct 
validity  of  the  items  and  scales.  However,  there  was  one  exception  to  this  finding: 
the  fit  of  the  model  for  the  questions  on  the  environment,  equipment,  and  supplies 
can  be  improved  by  allowing  for  covariance  (a  crossloading)  between  question 
number  27  (good  versus  poor  layout  of  the  work  place)  and  question  number  3 1 
(modern  versus  outdated  equipment). 

In  order  to  examine  convergent  validity,  we  compared  answers  to  the  various 
measures  across  the  three  job  categories  of  nurses,  physicians,  and  other  staff.  A 
multivariate  analysis  of  variance  showed  that  the  three  groups  were  different  on 
two  of  the  three  groups  of  variables:  measures  of  work  system  (Wilks’  lambda  = 
0.605,  P  <  0.001),  measures  of  perceived  performance  (Wilks’  lambda  =  0.994, 
not  significant),  and  measures  of  quality  of  working  life  (Wilks’  lambda  =  0.949, 
P  <  0.05).  For  the  measures  of  work  system,  all  measures  except  the  measure  of 
coordination  effectiveness  displayed  differences  between  the  three  job  categories. 
In  general,  physicians  had  more  positive  perceptions  of  the  work  system  than 
nurses  and  other  staff  (Figure  1).  As  for  quality  of  working  life,  univariate  tests 
were  statistically  significant  for  job  satisfaction  (P  <  0.05)  and  approached 
significance  for  fatigue  (P  =  0.06).  Physicians  were  more  satisfied  with  their  job 
and  reported  less  fatigue  than  nurses  and  other  staff. 
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Table  3.  Results  of  the  confirmatory  factor  analysis 


Model 

X2 

DF 

GFI 

AGFI 

CFI 

PGFI 

RMR 

RSMEA 

Communication  openness 

1.1 

1 

.998 

.981 

1.000 

.100 

.005 

0.02 

Communication  accuracy 

13.2 

1 

.978 

.782 

0.982 

.180 

.057 

.205 

Communication  timeliness 

- 

- 

- 

- 

- 

- 

- 

Communication  model 
with  one  factor 

111.4 

32 

.931 

.877 

.954 

.522 

.041 

.083 

Communication  model 
with  three  factors 

132.4 

36 

.921 

.859 

.940 

.504 

.060 

.096 

Coordination  mechanisms 

5.8 

4 

.992 

.970 

.993 

.265 

.018 

.040 

Workplace  design 

8.5 

6 

.990 

.966 

.995 

.283 

.030 

.038 

Equipment  design 

18.4 

7 

.979 

.936 

.982 

.326 

.036 

.075 

Access  to  supplies 

- 

- 

- 

- 

- 

- 

- 

- 

Design  model  with  one 
factor 

272.3 

73 

.886 

.836 

.870 

.616 

.090 

.097 

Design  model  with  three 
factors 

168.2 

69 

.926 

.887 

.935 

.608 

.070 

.070 

Unit  effectiveness 

5.8 

4 

.992 

.970 

.996 

.265 

.011 

.039 

Quality  of  working  life 

11.9 

4 

.984 

.940 

.984 

.262 

.019 

.083 

X2  (chi  square)  =  difference  between  model  and  data 

DF  =  degrees  of  freedom  (indication  of  the  complexity  of  the  model) 

GFI  =  goodness-of-fit  index 

AGFI  =  adjusted  goodness-of-fit  index  (fit  index  that  takes  model  complexity  into  account) 

CFI  =  comparative  fit  index  (fit  index  that  takes  sample  size  into  account) 

RSMEA  =  root  mean  square  error  of  approximation  (takes  error  of  approximation  in  the 
population  into  account) 

RMR  =  root  mean  square  residual  (represents  the  average  value  across  all  standardized 
residuals) 

Nonsignificant  X2  and  goodness-of-fit  indices  in  the  0.90s  accompanied  by  parsimonious  fit 
indices  in  the  0.50s  are  not  unexpected.  (Mulaik  SA,  James  LR,  Van  Alstine  J,  et  al.  Evaluation  of 
goodness-of-fit  indices  for  structural  equation  models.  Psychol  Bull  1989;105:430-45.) 


Predictive  validity  was  examined  by  conducting  a  correlation  analysis 
between  measures  of  the  work  system  and  the  measures  of  perceived  performance 
and  quality  of  working  life  (Table  3).  Results  showed  that  many  of  the  measures 
of  the  work  system  showed  a  statistically  significantly  correlation  with  most 
measures  of  perceived  performance  and  quality  of  working  life.  Communication 
openness,  time  pressure  affecting  patient  safety,  coordination  effectiveness, 
workplace  design,  and  equipment  design  showed  a  statistically  significant 
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Figure  1.  Comparison  of  three  job  categories  (nurses,  physicians,  and  other  staff) 
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correlation  with  both  measures  of  perceived  performance  and  the  three  measures 
of  quality  of  working  life.  The  other  measures  of  work  system  (except  for 
workload)  showed  correlation  with  the  measures  of  perceived  performance.  Job 
satisfaction  was  correlated  with  all  the  measures  of  work  system,  fatigue  was 
correlated  with  all  the  measures  of  work  system  except  communication 
timeliness,  and  tension  was  correlated  with  only  six  measures  of  work  system. 

We  also  conducted  a  series  of  stepwise  regression  analyses  with  each  of  the 
measures  of  perceived  performance  and  quality  of  working  life  as  the  dependent 
variables,  and  the  10  measures  of  work  system  as  independent  variables  (Table  4). 
The  measures  of  work  system  predicted  a  significant  amount  of  variance  of  unit 
effectiveness  (33  percent),  satisfaction  with  care  provided  (39  percent),  job 
satisfaction  (25  percent),  fatigue  (19  percent),  and  tension  (5  percent).  The 
measures  of  work  system  that  were  the  most  consistent  predictors  of  patient  safety 
were  communication  openness  and  time  pressure.  When  communication  was 
reported  to  be  open,  the  survey  participants  also  reported  high  unit  effectiveness, 
satisfaction  with  care  provided,  job  satisfaction,  and  low  tension.  High  time 
pressure  affecting  patient  safety  contributed  to  low  unit  effectiveness  and  job 
satisfaction,  and  high  fatigue. 
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Table  4.  Correlation  and  regression  analyses  between  measures  of  work  system 
(independent  variables)  and  perceived  performance  and  quality  of  working  life 
(dependent  variables) 


Perceived  performance 

Quality  of  working  life 

Work  system 

Unit 

effectiveness 

Satisfaction 
with  care 
provided 

Job 

satisfaction 

Fatigue 

Tension 

CORRELATION  ANALYSIS 

Communication 

openness 

.52*** 

42*** 

-.37*** 

2"|  *** 

-.18** 

Communication 

accuracy 

-.34*** 

-.32*** 

.30*** 

.16** 

.07 

Communication 

timeliness 

1  g*** 

22*** 

.23*** 

-.04 

-.06 

Time  pressure 
affecting  patient 
safety 

.39*** 

.59*** 

-.36*** 

-.26*** 

-.19** 

Workload 

-.03 

-.05 

.17** 

.25*** 

.13* 

Staffing 

-.14* 

-.12* 

2^*** 

.13* 

.10 

Coordination 

effectiveness 

.39*** 

.28*** 

-.31*** 

_  23*** 

-.15* 

Workplace 

design 

24*** 

.19*** 

-.35*** 

-.31*** 

-.19*** 

Equipment 

design 

zj  *** 

24*** 

^  1  *** 

-.20*** 

-.18** 

Access  to 
supplies 

.32*** 

24*** 

-.31*** 

2i  *** 

-.11 

STEPWISE  REGRESSION  ANALYSIS 

Adjusted  R2 

33%*** 

39%*** 

25%*** 

19%*** 

5%*** 

Significant 
predictors  (beta- 
coefficients) 

Comm,  open 
(.38) 

Supplies 

(.14) 

Coord. 

(.17) 

Pressure 

(.13) 

Pressure 

(.54) 

Comm,  open 
(.27) 

Workplace 

(-.13) 

Equipment 

(-.23) 

Pressure 

(-.19) 

Comm.  Open 
(-.17) 

Staffing 

(.14) 

Workplace 

(-.20) 

Workload 

(.21) 

Coord. 

(-.15) 

Pressure 

(-.14) 

Workplace 

(-.16) 

Comm. 

open 

(-.14) 

***p<0.001 


Discussion 

Data  reported  in  this  paper  provide  evidence  for  the  reliability  and  validity  of 
the  SEIPS  study’s  employee  questionnaire.  The  Cronbach’s  alpha  scores  for  all 
the  scales  except  two  were  above  0.70,  which  demonstrates  acceptable 
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reliability.10  Two  scales  had  Cronbach’s  alpha  scores  of  0.67,  very  close  to  the 
0.70  limit. 

The  confirmatory  factor  analysis  using  structural  equation  modeling  shows 
overall,  the  scales  of  the  work  system  (three  scales  on  communication,  one  scale 
on  coordination,  three  scales  on  environment/equipment/supplies),  the  scale  of 
unit  effectiveness,  and  the  measures  of  quality  of  working  life  demonstrated 
construct  validity.  A  few  results,  however,  provide  some  indication  that  construct 
validity  was  not  fully  achieved.  The  model  combining  the  three  scales  of 
communication  yielded  a  slightly  better  fit  of  the  data  as  compared  to  a  model 
keeping  the  three  scales  separate.  Because  our  systems  engineering  intervention 
focuses  on  various  aspects  of  communication,  we  decided  to  keep  the  three  scales 
of  communication  separate.  It  is  likely  that  the  intervention  affects  some  of  the 
communication  dimensions,  but  not  others.  Another  unexpected  result  concerns 
the  covariance  (a  cross-loading)  between  question  number  27  (good  versus  poor 
layout  of  the  work  place)  and  question  number  3 1  (modem  versus  outdated 
equipment).  This  result  is  difficult  to  interpret,  given  the  very  different  concepts 
tapped  by  those  two  questions. 

Convergent  validity  was  examined  by  comparing  responses  of  the  three  job 
categories  (nurses,  physicians,  and  other  staff)  on  three  groups  of  variables,  i.e., 
work  system,  perceived  performance,  and  quality  of  working  life.  We  found 
evidence  of  convergent  validity  for  the  measures  of  the  work  system  and  the 
measures  of  quality  of  working  life.  In  general,  physicians  reported  more  positive 
perceptions  of  the  work  system  and  higher  quality  of  working  life  than  nurses  and 
other  staff.  There  was  no  difference  between  the  three  job  categories  on  the 
measures  of  perceived  performance — i.e.,  unit  effectiveness  and  satisfaction  with 
care  provided.  The  lack  of  differences  between  nurses,  physicians,  and  other  staff 
on  these  measures  actually  demonstrates  their  similarity  in  reporting  information 
on  the  quality  and  safety  of  care  provided  by  their  surgery  centers,  as  well  as 
themselves. 

In  order  to  assess  predictive  validity,  we  examined  the  relationships  between 
the  work  system  and  perceived  performance  and  quality  of  working  life.  The 
measures  of  the  work  system  explained  a  significant  proportion  of  the  variance  for 
all  measures  of  perceived  performance  (33  percent  and  39  percent)  and  all 
measures  of  quality  of  working  life  (5-25  percent).  In  particular,  the  measures  of 
communication  openness  and  time  pressure  affecting  patient  safety  were  strong 
predictors  of  perceived  performance  and  quality  of  working  life.  The  systems 
engineering  intervention  implemented  in  our  SEIPS  project  tackles  the  issue  of 
communication  and  therefore  should  impact  one  of  these  two  important  work 
system  characteristics. 

A  major  weakness  of  the  data  used  to  examine  the  reliability  and  validity  of 
the  SEIPS  study  employee  questionnaire  is  the  cross-sectional  nature  of  the  study 
design.  All  data  were  collected  at  one  point  in  time;  the  researchers  were  not  able 
to  evaluate  relationships  over  time — in  particular,  relationships  between  work 
system  and  perceived  performance  and  quality  of  working  life.  However,  since 
our  SEIPS  study  involves  the  implementation  of  a  systems  engineering 
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intervention,7  we  will  be  able  to  see  whether  the  employee  questionnaire  can 
capture  changes  in  perceptions  of  work  system,  perceived  performance,  and 
quality  of  working  life.  This,  of  course,  assumes  that  the  systems  engineering 
intervention  is  successful  at  actually  implementing  changes  in  the  work  system 
and  processes. 

The  employee  questionnaire  provides  a  unique  opportunity  to  hear  from  health 
care  providers.  According  to  our  conceptual  framework,1  we  collected 
information  on  the  work  system,  perceived  performance  (including  unit 
effectiveness  and  satisfaction  with  care  provided),  and  quality  of  working  life.  We 
have  demonstrated  how  a  structured  method  (i.e.,  the  employee  questionnaire)  can 
be  developed  for  assessing  perceptions  and  opinions  from  the  outpatient  surgery 
centers. 


Conclusion 

Various  methods  can  be  used  to  gather  input  from  health  care  providers 
regarding  the  quality  and  safety  of  care  provided  by  their  organizations. 
Interviews,  focus  groups,  and  questionnaires  are  some  of  these  methods.  In  this 
study,  we  chose  to  use  a  questionnaire  approach  in  order  to  collect  structured, 
quantitative  information  on  the  work  system  and  various  outcomes  (e.g., 
perceived  unit  effectiveness  and  job  satisfaction).  The  data  presented  provide 
evidence  for  the  reliability  and  validity  of  our  employee  questionnaire.  Further 
steps  of  our  research  project  involve  the  implementation  of  a  systems  engineering 
intervention  that  will  be  evaluated  by  this  employee  questionnaire,  as  well  as  a 
patient  survey. 
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